The present invention relates to the field of computer software development and testing, and in particular, to a method for integrating automated software testing into a product as it is being developed to thereby simplify subsequent software product testing.
It has been common in the computer software arts that software product development was a distinct and separate process from software product testing. In the standard software product development lifecycle, development engineers would iteratively develop and refine a software product according to requirement or functional specifications. The development engineers frequently would test the product under development either in an ad-hoc manner or in a more rigorous manner. In either case, when the product was deemed to be completed by the development group of engineers, the software product was turned over to the testing process, usually a separate set of engineers from the group that developed the software product. Most, if not all, of the testing process utilized in the development phase would be discarded and the test engineers would begin anew evaluating the software product as against its corresponding product specifications.
In a typical test environment, the test engineers are provided with little or no internal information regarding the structure and design of the code comprising the software product. The testing effort would then proceed in a xe2x80x9cblack boxxe2x80x9d manner by applying test data (an external stimulus) and observing the output of the software product for proper response. In its crudest form, a test engineer determines (xe2x80x9cdreams upxe2x80x9d) potentially interesting test inputs and applies them to the software product while observing the output behavior of the software product for proper operation. This form of manual testing permits wide flexibility for the test engineer in creating input stimuli but provides little assistance to the test engineer in reproducing problems found during the test sequence. Manual test requires the test engineer to carefully note the sequence of test inputs which led to a specific problem test result.
Test engineers have developed or utilized a number of tools to assist in such xe2x80x9cblack boxxe2x80x9d testing. To automate the above discussed manual test process, scripting and macro languages are frequently used. The script/macro tool allows some degree of automation in the test sequence to aid in reproducing intermittent failures. A macro tool permits some added flexibility in the use of variables to customize the operation of the script based on externally supplied variable inputs. Some software products (e.g. word processing programs or spreadsheet programs) offer built-in macro features to reproduce sequences of user input keystrokes. Other program user interface (UI) environments (such as the Xwindows programming environment) provide for redirection of a program""s input from a script/macro file and thus may automate the testing of specific user keystroke sequences.
Simple xe2x80x9cBATxe2x80x9d files in the MS-DOS(copyright) environment or xe2x80x9cshell scriptsxe2x80x9d in the UNIX(copyright) environment are exemplary of such scripting tools used to partially automate the software product testing process. Macro facilities in the xe2x80x9cBATxe2x80x9d file or xe2x80x9cshell scriptxe2x80x9d interpreters and other macro replacement programs such as xe2x80x9cm4xe2x80x9d or xe2x80x9cperlxe2x80x9d in the UNIX(copyright) environment are exemplary of macro facilities used to enhance the scripting facilities in automating software product testing.
However, such scripting/macro based testing tools still provide little or no assistance to the test engineer in observing the software product output to automatically detect success and failure of each test. Collection and analysis of the test results remains largely a manual procedure. Correlation of the gathered test output with the timing and operation of the software product is difficult if not impossible.
An additional problem with scripting/macro tools arises in that the tool itself as well as the test scripts themselves must be ported to each computing environment in which the software product is to be tested. For example, a particular operation in a PC based Microsoft Windows(copyright) application may be tested by simulating an xe2x80x9cOPENxe2x80x9d menu operation while a similar function may require an xe2x80x9cOKxe2x80x9d menu operation in an Apple Macintosh(copyright) computing environment. The test scripts would have to change to test the same function in these two exemplary computing environments. In other words, the testing of underlying functions of an application may have to change as the user interface changes over a variety of computing environments. This creates an additional burden in porting and inaintaining the test tools and test scripts along with the software products. In addition, the porting and maintenance of the scripting/macro tools and test cases can be a source of additional errors which may be erroneously attributed to the failure of the software product under test.
Scripting/macro based test tools also tend to vary depending upon the external stimuli needs of each software product under test. The user interface for the test engineer using xe2x80x9cblack boxxe2x80x9d automated test tools tends to vary as the needs of each software product vary. Test engineers therefore must potentially learn a new user interface mode of operation for each software product to be tested.
All such xe2x80x9cblack boxxe2x80x9d testing methods, with or without the aid of automated scripting/macro testing tools, are typically performed without knowledge of the software product""s internal code structure. This lack of structural knowledge can make testing more cumbersome and time consuming. The lack of structural knowledge precludes certain styles of testing which may focus on particular error prone aspects of the software product revealed only by knowledge of the software product""s internal structure. Sometimes, for example, an error in one operation of the software product may not be externally observable in the output of the software product until subsequent operations are performed. The combination of the operations may eventually reveal the problem in an external manifestation, but the sequence of event may be lengthy back to the actual cause of the problem. Thus the use of external stimuli (xe2x80x9cblack boxxe2x80x9d) test methods, even in association with various automated test tools, does not permit the test engineer to exercise judgement with respect to problems more easily located through testing with knowledge of the software product""s internal code structure.
In addition, as software products evolve with ever increasing complexity, the burden of testing is being shifted more toward the development teams. It is simply impossible with xe2x80x9cblack boxxe2x80x9d testing techniques to exhaustively test all possible inputs (external stimuli) of many software products regardless of the size of the testing staff. Even testing of a large portion of the possible inputs is a difficult task in many cases. Therefore, the software development/testing lifecycle has begun to shift more of the testing efforts onto the responsibility of the development staff. For example, it is more frequent now that xe2x80x9cacceptance testingxe2x80x9d(testing for fundamental functionality) is made a part of the responsibility of the development staff. An added benefit of shifting some test burden to the development group is realized in the knowledge of internal code structure of the software product. The code structure knowledge of the development group may be applied to focus the testing effort on test sequences and scenarios most likely to create problems. In addition, the developers may construct the test cases to detect the failure in the test sequence as early as possible. It is a problem for developers to create a standardized test interface for the automated performance of test sequences in a manner that permits easy reproduction of the failed test sequences.
It is apparent from the above discussion that a need exists for an automated testing tool which permits a standardized interface for the generation and execution of test cases, which permits random sequences of testing, which permits easy reproduction of failed test sequences, and which automatically senses the success or failure of each test step.
The present invention solves the above identified problems and others to thereby advance the state of the useful arts by providing an easy to use, coding standard and user interface for a software developer to integrate testing of a software product into the development of the product. The testing application program interface (test API) is used by an application program developer to integrate a standardized test interface into the software product in its development phase. Use of the test API by the application program invokes standard functions in a dynamic linked library (DLL) to initiate a standard test user interface when the application program is run. The test tools are integrated into the application program and are therefore ported to any new computing environment by simply porting the application program to that environment. The development engineer, possibly in cooperation with the test engineers, thereby designs the test capability into the application program from its inception. These test hooks, combined with the standard, easy to use, graphical user interface of the test tools of the present invention, permit any engineer in the product development team to test the product. Development engineers may more effectively test the application program during its development phase to reduce the number of problems discovered later in the product""s lifecycle. Test engineers may apply the same easy to use graphical interface to more thoroughly test the functionality of the product.
The test API functions in the DLL provide a consistent, friendly, easy to use interface for testing of the application program. The DLL test functions may be operated in a random mode to generate pseudo-random test sequences. The DLL test functions automate recording of the pseudo-random seed and the corresponding sequence of test steps for easier reproduction of problems generated in the random sequence. The DLL functions can be used to automatically determine the success or failure of the corresponding test step by sensing a return value from the invocation of a corresponding sequence of execution in the application program. In addition, the application program may directly invoke other DLL functions to record an application failure. This alternative method is referred to herein as the assertion method.
The application programmer determines the operations to be tested in the application program and instantiates a test operation object (of a predefined test API object class) corresponding to each operation to be tested. The test operation object identifies an execute interface method (provided by the application programmer) to be invoked to execute the desired operation. Other interface methods of the test operation object identify functions which are to be performed to simulate the generation of input data and the acceptance of output data in response to operation of the execute interface method. For example, one interface method provides functions to be invoked for the manual or automatic generation of input data to test the operation. Another interface method of the test operation object provides functions to be invoked in response to I/O requests of the operation under test. Another interface method of the test operation object defines a set of rules to permit decisions to be, made in the invocation of the test object.
The application program is designed by the development engineers with the test operation objects defined as part of the application program. The application program is then compiled and linked with the test tool library (DLL) to provide the standard object class manipulation test methods of the present invention. By defining the test operation objects as discussed above, the application programmer has provided information (in the form of a collection of instantiated objects) useful to the test methods of the present invention to invoke each test operation and detect success or failure of each tested operation. Since the test operations are designed into the application program, and the library functions which implement the test operation classes of the present invention are available in a variety of computing environments, the application program may be more easily tested in many environments without the need to port unique test tools to each new computing environment. The test tools can be readily available in any computing environment to which the development staff chooses to port the application program. In addition, the test tools of the present invention, unlike scripting and macro methods of the past, do not require the development or testing teams to port the test cases between the various computing environments on which the program is to be tested. The test tools of the present invention are available for all platforms on which the application program may be run. The test suites are recorded in a portable manner by the test tools of the present invention in so-called playback files.
The application program is designed by the application programmer to invoke the test tool at an appropriate stage in the runtime-startup or processing of the application program. The test tool may then be operated in one of three modes: namely, random automated test operation selection, playback automated test operation selection, or manual playback creation mode.
In the random automated test selection mode, the test methods of the present invention pseudo randomly select from among the set of instantiated test operation objects defined by the application programmer. As each test operation object is randomly selected, the execute function method associated with the object is invoked to perform the developer""s designed test function. The random selections are logged in a playback file to permit later reproduction of a failed sequence. In addition, the results of each execute function method invocation are logged in a file by the methods of the present invention so that a sequence may be reproduced if it produces a problem in the application program under test. This process continues randomly selecting test operation objects for which the corresponding execute function is invoked until directed to stop by a user or other termination conditions are met.
In the playback automated test operation, the test operation selections made in a previous test sequence are replayed from the playback file created during an earlier performance of the test sequence. This mode of operation is used, for example, to permit reproduction of test sequences which led to erroneous results in previous invocations.
The manual playback creation mode of operation permits a user of the present invention to create or edit a playback file. This mode allows the user finer control over the test sequencing to more rapidly debug an application program.
The methods of the present invention enable automation of the testing of an application program while integrating the test cases with the program development. This integration of the test case and program development permits more thorough analysis and regression of the application program. The automation aspects of the test methods of the present invention permit easier, unattended test operation to more effectively test an application program.